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(54) INTERFACE DEVICE 
(57)Abstract 

PROBLEM TO BE SOLVED: To improve operability by easily 
operating equipment without requiring any input device such as 
keyboard or mouse by displaying the feature of shape of a hand 
recognized by a recognizing means on a screen as a special shape. 
SOLUTION: When a user projects one finger while facing equipment 
provided with an interface device, the icon of numeral '1* is 
displayed on a display 2 and the television of 1st menu 201 is 
displayed while being emphasized. At such a time, it is also available 
to output a sound or a voice from the display device 2 
corresponding to the emphasized indication. In this case, when the 
user changes the shape of the hand to project two fingers, the icon 
of numeral '2' is displayed on the display 2 and the network of 2nd 
menu 202 is displayed while being emphasized. By keeping the same 
shape of the hand for a fixed time in such a state, for example, the 
2nd menu 202 is selected and it can be commanded to a host 
computer 1 to perform the display of network terminal. 
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* NOTICES* 

JPO and NCIPI are not responsible for any 
damages caused by the use of this translation. 

l.This document has been translated by computer. So the translation may not reflect the 
original precisely. 

2 **** s hows the word which can not be translated. 
3.1n the drawings, any words are not translated. 

CLAIMS 



[Claim(s)] 

[Claim l] The interface device equipped with a recognition means to recognize the 

configuration of an operator's hand, a display means to display on a screen by making into a 

special configuration the description of the configuration of the hand recognized by said 

recognition means, and the control means that controls the information displayed in said 

screen by the special configuration displayed on the screen by said display means. 

[Claim 2] The interface device according to claim 1 which a recognition means displays on a 

screen by making into a special configuration the configuration of the hand that have 

recognized the motion of said hand with the configuration of a hand, and the display means 

has been recognized by said recognition means, and the description of a motion. 

[Claim 3] The interface device according to claim 1 controlled so that a control means chooses 

the information displayed in the screen. 

[Claim 4] The interface device characterized by providing the following It is the image pick-up 
section at least, the configuration of the body in the picturized image and - or the 
recognition section of operation which recognizes a motion the configuration of the body 
recognized by said recognition section of operation - and - or the frame memory which saves 
the image which was equipped with the display which displays a motion and was picturized in 
said image pick-up section Image difference operation part which prepares the criteria image 
memory which accumulates the image picturized to the time amount before the image saved 
in said frame memory as a criteria image, and extracts the difference between the image in 
said frame memory, and the criteria image accumulated into said criteria image memory in 
said recognition section of operation 

[Claim 5] The interface device according to claim 4 which prepared the renewal section of a 
criteria image which updates the criteria image accumulated into criteria image memory in a 
new image. 

[Claim 6] The interface device according to claim 4 which formed the timer which calculates 

spacing of renewal of a criteria image in the renewal section of a criteria image. 

[Claim 7] the configuration of the hand of the user in the image picturized with the image 
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pick-up section at least - and - or with the recognition section of operation which recognizes a 
motion the configuration of the hand of the user who has recognized by said recognition 
section of operation -- and - or it having the display which displays a motion and with a 
profile extract means to extract a user's picturized profile The profile wave operation part 
which pursues said profile and calculates the relation between the include angle of a border 
line, and the die length of a border line, i.e., a profile wave, The interface device which filtered 
the profile wave calculated by said profile wave operation part, prepared the configuration 
filter which generates the configuration wave showing a predetermined configuration, and 
constituted said recognition section of operation. 

[Claim 8] The interface device according to claim 7 it was made to make actuation of a user 
judge based on the configuration wave which constituted two or more configuration filters 
with two or more band pass filters with which bands differ, and was generated with said two 
or more configuration filters. 

[Claim 9] The interface device according to claim 7 which constituted two or more 
configuration filters at least with the band pass filter of the profile wave configuration 
corresponding to the irregularity of a hand, and the band pass filter of the profile wave 
configuration corresponding to the irregularity of a finger. 

[Claim 10] The interface device according to claim 7 which prepared the coordinate of the 
profile configuration of the user who picturized, and the coordinate table which makes 
correspondence with the profile wave calculated by profile wave operation part memorize, 
prepared the coordinate operation part which calculates the coordinate in which the 
predetermined configuration in an image pick up image exists using the wave height 
existence location and said coordinate table of a configuration wave, and constituted the 
recognition section of operation. 

[Claim ll] The interface device according to claim 7 prepare [ interface device ] the 
configuration judging section which counts the pulse number in the configuration wave 
generated with the configuration filter, constitute [ interface device ] the recognition section of 
operation, and it was made to make an objective configuration judge with the output value of 
said configuration judging section. 

[Claim 12] The interface device according to claim 7 which formed the differentiator which 
differentiates the configuration wave generated with the configuration filter, and constituted 
the recognition section of operation. 

[Claim 13] The interface device characterized by providing the following Display means An 
input means to change the location and configuration of cursor which are displayed on said 
display means A cursor storage means to memorize the configuration of the representation 
point coordinate representing the location of said cursor, and said cursor A body storage 
means to memorize the configuration of the representation point coordinate representing the 
location of display objects other than said cursor, and said display body, The location and 
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configuration of a display object which are memorized for the location and the configuration, 
and said body storage means of the cursor memorized for said cursor storage means are used. 
It has an interaction judging means to judge the interaction between said cursor and said 
display objects. Said interaction judging means A distance count means to calculate the 
distance between the representation point of said cursor of at least one point, and the 
representation point of said display object of at least one point, A comprehensive judgment 
means to opt for the interaction of said cursor and said display object using the recognition 
result of a recognition means of operation to recognize migration of said cursor, or change of a 
configuration, and the distance which said distance count means calculates and said 
recognition means of operation 

[Claim 14] The interface device according to claim 13 characterized by an interaction judging 
means generating an interaction to the display object below the criteria which the distance 
which a distance count means calculates defined beforehand when the recognition means of 
operation has recognized the actuation registered beforehand. 

[Claim 15] The interface device according to claim 13 characterized by for an interaction 
judging means to opt for the interaction of said cursor and display object based on the 
migration direction of the cursor which establishes a movement vector calculation means to 
compute the migration direction and movement magnitude of cursor in an operating space, 
and constitutes an interaction judging means, and said migration direction calculation means 
computes, and the movement magnitude of cursor. 

[Claim 16] The interface device according to claim 15 characterized by an interaction judging 
means generating an interaction when the movement magnitude of the cursor which a 
movement vector calculation means computes is below the reference value defined beforehand. 
[Claim 17] The interface device according to claim 15 characterized by an interaction judging 
means generating an interaction to the display object which exists near the production of the 
migration direction of the cursor which a movement vector calculation means computes. 
[Claim 18] The interface device according to claim 13 characterized by an interaction judging 
means generating an interaction when the configuration of cursor and the configuration of a 
display object become the combination registered beforehand. 

[Claim 19] The interface device according to claim 13 characterized by an interaction judging 
means generating an interaction about the case where the configuration of cursor and the 
configuration of a display object which are equipped with a configuration judging means to 
recognize a cursor configuration and the configuration of a display object, and constitute an 
interaction judging means, and said shape -recognition means recognizes are in agreement. 
[Claim 20] The interface device according to claim 13 characterized by an interaction judging 
means generating an interaction when the actuation which the recognition means of 
operation registered beforehand has been recognized to the display object near the production 
top of the look which establishes a look input means to detect the direction of a look, and said 
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look input means detects. 

[Claim 21] The interface device according to claim 20 characterized by an interaction judging 
means generating an interaction when the actuation which cursor existed near the production 
top of said look, and the recognition means of operation registered beforehand has been 
recognized to the display object near the production top of the look which a look input means 
detects. 

[Claim 22] The interface device according to claim 13 characterized by establishing a study 
means to learn the physical relationship of cursor and the target display object, the 
configuration of said cursor, and the configuration of said display object, and an interaction 
judging means opting for an interaction based on the study result of a study means when an 
interaction is generated. 

[Claim 23] The interface device according to claim 22 characterized by an interaction judging 
means generating an interaction when the physical relationship of cursor and the target 
display object or the configuration of said cursor, and the configuration of said display object 
are similar with the physical relationship or the configuration which the study means learned 
in the past. 

[Claim 24] The interface device according to claim 13 characterized by establishing a 
coordinate transformation means to perform coordinate transformation to the input from the 
cursor storage section and the body storage section to a distance count means, and 
constituting an interaction judging means. 

[Claim 25] The interface device according to claim 24 characterized by performing coordinate 
transformation so that the physical relationship of cursor and the target display object may be 
made to approach when an interaction is generated. 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to the interface device which outputs and 
inputs the device which has the display of information machines and equipment, such as a 
computer and a word processor, television, etc. 
[0002] 

[Description of the Prior Art] There are some which are performed by displaying cursor on the 
coordinate location detected with the mouse in the display screen although add some another 
information to the information in a display, or the information on a display is changed into it 
as this kind of a conventional interface device or being chosen. 

[0003] Drawing 30 shows the outline of this kind of conventional interface device. In drawing 
30 , 501 is a host computer, 502 is a display, and the manual operation button 503,504,505 of 
imagination is displayed with the host computer 501 during the display 502. 506 is a mouse 
cursor, and based on the movement magnitude of the mouse 507 detected with the mouse 507, 
a host computer 501 carries out a display control so that it may move all over a screen 
synchronizing with a motion of a mouse 507. By moving a mouse 507, by moving a mouse 
cursor 506 to the location of the manual operation button of imagination of the arbitration in 
a display screen, and pushing the switch 508 on a mouse 507, a user chooses a manual 
operation button and can give directions of operation to a host computer 501 now. 
[0004] 

[Problem(s) to be Solved by the Invention] However, with the above-mentioned conventional 
configuration, apart from the body of a device, an input device called a mouse is required, and 
the pedestal of the table which operates a mouse is required, and using for a portable 
information device etc. is not suitable. Moreover, in order to operate it through a mouse, it is 
not necessarily a direct and intelligible interface. 

[0005] In view of this point, this invention does not need input units, such as a keyboard and a 
mouse, but actuation of a device is simply possible for it, and its operability is further more 
good, and it tends to offer a direct and intelligible interface device. 
[0006] 

[Means for Solving the Problem] This invention with the special configuration displayed on 
the screen by recognition means to recognize the configuration of an operator's hand, display 
means to display on a screen by making into a special configuration the description of the 
configuration of the hand recognized by this recognition means, and this display means It is 
the interface device equipped with the control means which controls the information 
displayed in the screen, and the information displayed on the screen can be controlled only by 
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changing the configuration of a hand. 

[0007] Furthermore, the good interface means of operability is further offered by recognizing 
a motion of a hand. The criteria image memory which accumulates the image picturized to the 
time amount before the image saved in the frame memory which saves the image which 
picturized the configuration of a hand or the motion at the motion recognition, and said frame 
memory as a criteria image is prepared, and it attains by extracting the difference between 
the image in a frame memory, and the criteria image accumulated into criteria image memory, 
moreover, the configuration of the hand of the user in the image picturized as other 
recognition approaches - and - or the motion was extracted as a user's profile, the profile was 
pursued, and it has attained by generating the configuration wave which calculates and 
filters the relation between the include angle of a border line, and the die length of a border 
line, i.e., a profile wave, and expresses a predetermined configuration. 

[0008] Furthermore, the cursor display means for displaying on a screen by making the 
description of the configuration of a hand into a special configuration, and operating it as 
cursor, A means to memorize relation with display objects other than the cursor display as the 
representation point coordinate and the configuration of representing the location of display 
objects other than a cursor display, The means which carries out the operation judging of the 
interaction between a cursor display and said display object is established, and when 
imagination manipulator-izes a cursor display and it displays it, and holding the displayed 
virtual body, actuation can be smoothly realized by the interaction in alignment with an 
intention of an operator. 

[0009] If a user tends toward a recognition means, for example, the interface device 
constituted as mentioned above holds up a hand, the icon display for the so-called menu 
manipulation will be made in a screen, and the special configuration equivalent to the 
configuration of a hand will become controllable [ according to the icon display ]. 
[0010] Or if directions are given according to a gesture, while the special configuration in 
which the given gesture was set up on the display screen corresponding to the configuration of 
a hand will be displayed Choose the virtual switch which the motion was also displayed, for 
example, was displayed on the display screen according to a gesture, or [ for the purpose of the 
display object displayed on the screen ], it cannot hold, cannot carry, or cannot carry out, and 
input units, such as a mouse, are not needed, but actuation of a very simple device is attained. 
[0011] Moreover, the special configuration setup corresponding to the configuration of a hand 
operates as a virtual manipulator in addition to mere cursor, and the interface whose 
operability improved further can be realized by making serially the judgment which suited 
the actuation intention of an operator automatically in an interaction with a display object 
operating with a virtual manipulator. . 
[0012] 

[Embodiment of the Invention] 
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(Gestalt of the 1st operation) The 1st operation gestalt of this invention A display means to 
display the description of the configuration of the hand recognized by a recognition means and 
these recognition means, such as image pick-up equipment, in order to recognize the 
configuration of an operator's hand on a screen as a special configuration by an icon etc,, It is 
the interface device equipped with the control means which controls the information which 
special configurations, such as an icon displayed on the screen by this display means, were 
operated as the so-called cursor, and was displayed in the screen by changing the 
configuration of a hand. 

[0013] (Gestalt of the 2nd operation) The 2nd operation gestalt of this invention the 
configuration of the body in the image picturized with the image pick up section at least and 
-- or with the recognition section of operation which recognizes a motion the configuration of 
the body recognized by this recognition section of operation - and -* or the display which 
displays a motion constituting and with the frame memory which saves the image picturized 
in said image pick-up section The criteria image memory which accumulates the image 
picturized to the time amount before the image saved in this frame memory as a criteria 
image is prepared. In the recognition section of operation The image in a frame memory, It is 
the interface device which prepared the image change extract section which extracts a 
difference with the criteria image accumulated into criteria image memory. 
[0014] (Gestalt of the 3rd operation) The gestalt of operation of the 3rd of this invention the 
configuration of the hand of the user in the image picturized with the image pick-up section at 
least - and - or with the recognition section of operation which recognizes a motion the 
configuration of the hand of the user who has recognized by this recognition section of 
operation and or it having the display which displays a motion and with a profile extract 
means to extract a user's picturized profile The profile wave operation part which pursues 
this extracted profile and calculates the relation between the include angle of a border line, 
and the die length of a border line, i.e., a profile wave, It is the interface device which filtered 
the profile wave calculated by this profile wave operation part, prepared the configuration 
filter which generates the configuration wave showing a predetermined configuration, and 
constituted said recognition section of operation. 

[0015] If a user goes to the image pick-up section of the interface device constituted as 
mentioned above, for example, directions are given according to a gesture, the image pick up 
section will picturize a user's image. A profile extract means extracts the profile of a user's 
image, and this profile is changed as the include angle of the border line to a horizontal line, 
i.e., a profile wave, by setting an axis of abscissa as the die length of the border line which 
made the reference point on a profile the start point by profile wave operation part. This 
profile wave is changed into the configuration wave which expresses the shape of toothing of a 
finger with the configuration filter constituted with the band pass filter of a predetermined 
band, for example, the band pass filter equivalent to the irregularity of a finger, while it 
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calculates a location with a hand, only counts the pulse number which exists in this 
configuration wave, and can make the number of the finger currently sent, i.e., the 
configuration of a hand, judge correctly. The virtual switch which the given gesture was 
displayed on the display screen of a display based on a location and a configuration of this sort, 
for example, was displayed on the display screen cannot be chosen according to a gesture, and 
input units, such as a mouse, are not needed, but actuation of a very simple device is attained. 
[0016] Moreover, two or more band pass filters with which bands differ may constitute two or 
more configuration filters, and based on the configuration wave generated with said two or 
more configuration filters, you may constitute so that actuation of a user may be made to 
judge. Thereby, two or more configurations can be recognized now. 

[0017] Moreover, the band pass filter of the profile wave configuration corresponding to the 
irregularity of a hand and the band pass filter of the profile wave configuration corresponding 
to the irregularity of a finger may constitute two or more configuration filters at least. It is 
changed into the smooth configuration wave only reflecting the irregularity of the part of a 
hand by this, and is changed into the configuration wave only reflecting the irregularity of 
**** 

[0018] Moreover, the coordinate of the profile configuration of the user who picturized, and the 
coordinate table which makes correspondence with the profile wave calculated by profile wave 
operation part memorize may be prepared, the coordinate operation part which calculates the 
coordinate in which the predetermined configuration in an image pick-up image exists may be 
prepared using the wave height existence location and said coordinate table of a configuration 
wave, and the recognition section of operation may be constituted. Thereby, the coordinate of 
a profile configuration calculates and the coordinate is outputted. 

[0019] Moreover, the configuration judging section which counts the pulse number in the 
configuration wave generated with the configuration filter is prepared, the recognition section 
of operation is constituted, and you may make it make an objective configuration judge with 
the output value of said configuration judging section. The judgment of whether to be the 
configuration which grasped the hand for whether it is the configuration of 2 fingers with the 
pulse number by this is made easily. 

[0020] Moreover, the differentiator which differentiates the configuration wave generated 
with the configuration filter may be formed, and the recognition section of operation may be 
constituted. By differential, a wave becomes pulse-like more and the count of a pulse number 
becomes easy. 

[0021] (Gestalt of the 4th operation) Although the gestalt of old operation has described the 
example which operates it about the secondary screen displayed on the display screen, it 
requires the gestalt of this operation for the actuation at the time of displaying an 
imagination three dimension on the two-dimensional display screen. In the imagination 
three-dimension space generally displayed, if the actuation which holds the virtual body in a 
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virtual space using cursor is assumed, it will become the following configurations. 
[0022] For an input unit and A2, in drawing 21 , the cursor coordinate storage section and A3 
are [ Al / a display and A5 of the body coordinate storage section and A4 ] the contact decision 
sections. Drawing 22 shows the cursor of the manipulator configuration of two fingers which 
can be displayed from the configuration of an operator's hand like the gestalt of the already 
described operation. The condition that the condition to which drawing 22 (a), (c), and (e) 
opened the finger, drawing 22 (b), (d), and (f) closed the finger is shown. Drawing 23 shows the 
example of the virtual body in a virtual space. An operator assumes the actuation which holds 
the virtual body in virtual three -dimension space using the cursor of two fingers here. 
Drawing 24 (a) and drawing 24 (b) show arrangement of the cursor in the virtual space when 
holding a virtual body using cursor, and a virtual body. Drawing 25 shows the display of 
display A4. 

[0023] If actuation of an operator is given to the input section Al, spacing of the cursor 
coordinate memorized in the cursor coordinate storage section A2 and two fingers of cursor 
will be updated based on said actuation. Display A4 draws the virtual space which contains 
cursor and a virtual body using the positional information of the virtual body which the 
information which the cursor coordinate storage section A2 memorizes, and body coordinate 
storage section A3 memorize. Here, cursor and a virtual body calculate whether it contacts in 
a virtual space using the positional information of the virtual body which the information the 
cursor coordinate storage section A2 remembers contact decision section A5 to be, and body 
coordinate storage section A3 memorize. If the distance in the virtual space between two or 
more fields which constitute cursor and a virtual body is specifically calculated about between 
each field and a virtual body contacts between two fingers of cursor, it will judge that cursor 
has held the virtual body and an objective coordinate will be henceforth changed in 
accordance with migration of cursor. 

[0024] However, by such technique, even when in arrangement as shown in drawing 24 (a) 
and (b) the display by the display becomes like drawing 25 and the location of cursor and a 
virtual body is not completely in agreement in a virtual space, an operator may 
incorrect- judge that the coordinate is in agreement. Moreover, it becomes difficult [ a display 
which wrote together the case where three-dimensional display equipment is used, drawing 
24 (a), and (b) ] from the difference in the depth perception in real space and a virtual space 
smooth to operate it. 

[0025] Thus, the interaction (a virtual body is held in the aforementioned example) which met 
the intention of an operator in the interaction of cursor and the virtual body in a virtual space 
can be smoothly realized neither according to the difference between the depth perception in 
the imagination space which is an operating space, and the depth perception in real space, nor 
the difference between a motion of the cursor which an operator means, and a motion of 
actual cursor (for example, when holding a virtual body with the manipulator of imagination 
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etc.). 

[0026] With the gestalt of this operation, while operability is improved by cursor control in 
non-contact [ with a gesture etc. ] in an operator in a virtual space Only the distance between 
the cursor in a virtual space and the component (it is a front face when a virtual space is a 
three dimension) of a virtual body does not determine the existence of generating of an 
interaction with the virtual body in a virtual space. The judgment of whether cursor causes a 
virtual body and an interaction can be brought close to an intention of an operator, and the 
operability of an interface can be further raised because an interaction judging means causes 
an interaction also to the object with which the distance in a virtual space is not necessarily 
close. Furthermore, it -becomes possible to cause an interaction also to the object with which 
the distance in a virtual space is not necessarily close. 

[0027] An input means by which the 1st configuration of the gestalt of this operation changes 
the location and configuration of cursor of displaying on a display means and said display 
means, A cursor storage means to memorize the configuration of the representation point 
coordinate representing the location of said cursor, and said cursor, A body storage means to 
memorize the configuration of the representation point coordinate representing the location of 
display objects other than said cursor, and said display body, The location and configuration of 
a display object which are memorized for the location and the configuration, and said body 
storage means of the cursor memorized for said cursor storage means are used. It constitutes 
from an interaction judging means to judge the interaction between said cursor and said 
display objects. Said interaction judging means A distance count means to calculate the 
distance between the representation point of said cursor of at least one point, and the 
representation point of said display object of at least one point, It is the interface device 
characterized by consisting of a recognition means of operation to recognize migration of said 
cursor, or change of a configuration, and a comprehensive judgment means to opt for the 
interaction of said cursor and said display object using the recognition result of the distance 
which said distance count means calculates, and said recognition means of operation. 
[0028] By this configuration, the existence of generating of the interaction of the cursor which 
an operator operates in a virtual space, and the virtual body in a virtual space Only the 
distance between said cursor in a virtual space and the component Gt is a front face when a 
virtual space is a three dimension) of said virtual body does not determine. When a 
comprehensive judgment means judges the existence of generating of an interaction by 
actuation of the cursor which the distance and the recognition means of operation between 
the representation points which a distance count means computes recognize, it becomes 
possible to cause an interaction also to the object with which the distance in a virtual space is 
not necessarily close. 

[0029] Moreover, in the 1st configuration, when the actuation which the recognition means of 
operation registered beforehand has been recognized, it is good also as the 2nd configuration 
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an interaction judging means generates [ configuration ] an interaction to the display object 
below the criteria which the distance which a distance count means calculates defined 
beforehand. 

[0030] Moreover, it is good also as the 3rd configuration whose interaction judging means opts 
for the interaction of said cursor and display object based on the migration direction of the 
cursor which sets in the 1st and 2nd configuration, establishes a movement-vector calculation 
means compute the migration direction and the movement magnitude of cursor in an 
operating space, and constitutes an interaction judging means, and said migration direction 
calculation means computes, and the movement magnitude of cursor. 

[0031] Moreover, in the 3rd configuration, when the movement magnitude of the cursor which 
a movement vector calculation means computes is below the reference value defined 
beforehand, an interaction judging means is good also as the 4th configuration which 
generates an interaction. 

[0032] Moreover, in the 3rd and 4th configuration, it is good also as the 5th configuration an 
interaction judging means generates [ configuration ] an interaction to the display object 
which exists near the production of the migration direction of the cursor which a movement 
vector calculation means computes. 

[0033] Moreover, in the 1st - the 5th configuration, when the configuration of cursor and the 
configuration of a display object become the combination registered beforehand, an 
interaction judging means is good also as the 6th configuration which generates an 
interaction. 

[0034] Moreover, in the 1st ■ the 6th configuration, an interaction judging means is good also 
as the 7th configuration which generates an interaction about the case where the 
configuration of cursor and the configuration of a display object which are equipped with a 
configuration judging means to recognize a cursor configuration and the configuration of a 
display object, and constitute an interaction judging means, and said shape-recognition 
means recognizes are in agreement. 

[0035] Moreover, in the 1st - the 7th configuration, when the actuation which the recognition 
means of operation registered beforehand has been recognized to the display object near the 
production top of the look which establishes a look input means to detect the direction of a 
look, and said look input means detects, an interaction judging means is good also as the 8th 
configuration which generates an interaction. 

[0036] Moreover, in the 8th configuration, when the actuation which cursor existed near the 
production top of said look, and the recognition means of operation registered beforehand has 
been recognized to the display object near the production top of the look which a look input 
means detects, an interaction judging means is good also as the 9th configuration which 
generates an interaction. 

[0037] Moreover, in the 1st • the 9th configuration, when an interaction is generated, it is good 



JP09- 185456 
Translated by Computer 

12 

also as the 10th configuration whose interaction judging means establish a study means to 
learn the physical relationship of cursor and the target display object, the configuration of 
said cursor, and the configuration of said display object, and opts for an interaction based on 
the study result of a study means. 

[0038] Moreover, in the 10th configuration, when the physical relationship of cursor and the 
target display object or the configuration of said cursor, and the configuration of said display 
object are similar with the physical relationship or the configuration which the study means 
learned in the past, an interaction judging means is good also as the 11th configuration which 
generates an interaction. 

[0039] Moreover, in the 1st - the 11th configuration, it is good also as the 12th configuration 
which establishes a coordinate transformation means to perform coordinate transformation to 
the input from the cursor storage section and the body storage section to a distance count 
means, and constitutes an interaction judging means. 

[0040] Moreover, in the 12th configuration, when an interaction is generated, it is good also as 
the 13th configuration which performs coordinate transformation so that the physical 
relationship of cursor and the target display object may be made to approach. 
[0041] (The 1st example) Drawing 1 shows the appearance of the 1st example of the interface 
device by this invention. In drawing 1 , the display which uses 1 for a host computer and uses 
2 for a display, and 3 are CCD cameras which picturize an image. CCD camera 3 arranges the 
image pick- up side in the same direction as the display direction of a display 2, and when a 
user goes to the screen, it can picturize the shape of a user's note. On a display 2, the icon 200 
which performs the display reflecting a menu 201,202 and the shape of said note can be 
displayed now. 

[0042] Drawing 2 shows the detail block diagram of this example. The image inputted from 
CCD camera 3 is stored in a frame memory 21. In the criteria image memory 25, the image of 
the background which does not contain the person picturized beforehand is saved as a criteria 
image. A criteria image can be updated at any time if needed. 

[0043] While the configuration discernment means 22 removes a background image out of an 
image by extracting the difference of the image stored into the frame memory 21, and the 
image saved in the criteria image memory 25 for example, it is the configuration of 1 finger as 
extracts the part equivalent to a user's hand and the configuration shows to drawing 3 (a) - 
moreover, it is the configuration of 2 fingers as shown in drawing 3 (b) - or it judges whether 
other than this, it is drawing 3 (c). 

[0044] Drawing 4 is what showed the gestalt of detail operation of the configuration 
discernment means 22, and shows the example constituted by the image difference operation 
part 221, the profile extract section 222, and the configuration discernment section 223. 
Image difference operation part 221 As mentioned above, the difference of the image stored 
into the frame memory and the image saved in criteria image memory is calculated. Thereby, 
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the parts of a body, for example, a user, to extract and a background are separable. For 
example, image difference operation part 221 When constituted from a simple subtraction 
circuit, as shown in drawing 5 , only the part of the hand of the user in the image in a frame 
memory can be extracted. The profile extract section 222 is the image difference operation 
part 221. The profile configuration of the body which exists in the image of the calculated 
result is extracted. As a concrete example of an approach, a profile configuration can be easily 
extracted by extracting the edge of an image. 

[0045] It judges whether the configuration discernment section 223 is the configuration of 2 
fingers where it is shown in drawing 3 (b) again whether it is the configuration of 1 finger as 
carries out detail discernment of the profile configuration of a hand, for example, the 
configuration shows to drawing 3 (a) extracted by the profile extract section 222. It is possible 
to use template matching, the matching technique with a geometric model, a neural network, 
etc. as a method of identifying a configuration. 

[0046] The icon generation section 24 is the configuration discernment section 223. Based on 
the discernment result of the configuration of the hand to depend, an icon image is generated 
as a special configuration which should be displayed on a display, for example, - a hand - a 
configuration - discernment - a result one - a ** - a finger - a configuration -- it is - if 
for example - drawing 6 (-- a --) - being shown as a figure - " - one - " - an icon - 
generating * moreover - two - a ** a finger - a configuration - it is - if for example 
drawing 6 - (-- b •-) ■■ being shown — as - a figure - " - two - " - an icon - generating . A 2 
finger type icon if it is the configuration of 2 fingers again, as shows a 1 finger type icon if the 
discernment result of the configuration of a hand is the configuration of 1 finger as a 
configuration of an icon, as shown in drawing 6 (c) to drawing 6 (d) etc. may be generated. A 
display and control section 23 is the configuration discernment section 223. A display is 
controlled based on the discernment result of the configuration of the hand to depend. For 
example, while displaying said icon based on a discernment result, it displays by emphasizing 
the menu beforehand matched with the discernment result based on a note-like discernment 
result. 

[0047] The example of actuation by this example constituted as mentioned above is shown 
below. If a user faces to the device equipped with the interface device of this example and a 
hand is made into the configuration of 1 finger, as shown in drawing 7 (a), while the icon of 
figure" 1" will be displayed on a display, highlighting of the display of television which is the 
1st menu is carried out. It is possible to call an operator's attention further with outputting a 
sound or voice from a display unit in accordance with highlighting at this time. Here, if a 
hand is made into the configuration of 2 fingers like drawing 7 (b), while the icon of figure"2" 
will be displayed on a display, highlighting of the display of the network which is the 2nd 
menu is carried out. The 2nd menu is chosen by maintaining the configuration of the same 
hand for example, during 1 scheduled time in this condition, and a command can be given so 
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that a network terminal may be displayed on a host computer. Voice etc. may be used together 
for selection of a menu. In except the configuration of the hand beforehand decided like 
drawing 7 (c), an icon and a menu are not displayed on a display, and an instruction is not 
sent to a host computer. 

[0048] Thus, according to this example, it is possible to identify the configuration of the hand 
in the picturized image and to control devices, such as a computer, according to a discernment 
result, and it is possible to operate it by non-contact from remoteness, without using devices, 
such as a keyboard and a mouse. Moreover, positive actuation is attained that it can be 
operated while a user checks a discernment result by making the result of having identified 
the configuration of a hand reflect in a screen, and it is easy to use. 

[0049] In addition, although this example showed the example only applied to menu selection, 
when an icon display sets ****** as a configuration with a hand beforehand so that a picture, 
a text, etc. may be replaced, it cannot be overemphasized that it is also easy to write a picture 
and an alphabetic character in a display and to control them. 

[0050] (The 2nd example) Drawing 8 shows the appearance of the 2nd example of the 
interface device by this invention. In drawing 8 , the same numeric value shows a component 
to the part which has the same function as the 1st example. That is, the display which uses 1 
for a host computer and uses 2 for a display, and 3 are CCD cameras which picturize an image. 
CCD camera 3 arranges the image pick-up side in the same direction as the display direction 
of a display 2, and when a user goes to the screen, it can picturize a user's gesture. On the 
screen of a display 2, the icon 203 of the arrow-head cursor which chooses the_virtual switches 
204, 205, and 206 and said virtual switch can be displayed now. 

[0051] Drawing 9 shows the block diagram of the detailed configuration of the gestalt of this 
operation. The image inputted from CCD camera 3 is stored in a frame memory 21. In the 
criteria image memory 25, the image picturized beforehand is saved as a criteria image. A 
timer 261 and the renewal section 262 of an image constitute the renewal section 26 of a 
criteria image, it transmits the newest image saved at the frame memory 21 with the fixed 
time interval directed with a timer 261 to the criteria image memory 25, and updates a 
criteria image. 

[0052] It judges whether the recognition section 22 of operation is the configuration of a fist 
where the part which is equivalent to a user's hand, for example is extracted, and the 
configuration shows drawing 10 (b) whether it is the configuration of 1 finger as shown in 
drawing 10 (a) again while removing a background image out of an image by extracting the 
difference of the image stored into the frame memory, and the image saved in criteria image 
memory. 

[0053] Drawing 11 is what showed the detail of the recognition section 22 of operation, and 
shows the example constituted by the image difference operation part 221, the profile extract 
section 222, the form status change ized discernment section 225, and the location detecting 
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element 224. 

[0054] The image difference operation part 221 calculates the difference of the image stored 
into the frame memory 21 as mentioned above, and the image saved in the criteria image 
memory 25. While the part of the hand of a body, for example, a user, to extract as a motion 
and the part of a background are separable by this, only the body image which is moving can 
be extracted. For example, when the image difference operation part 221 is constituted from a 
simple subtraction circuit, as shown in drawing 12 , only the part of the hand in a criteria 
image and the part of the hand in the newest image in a frame memory can be extracted, and 
only the part of the hand which is moving can be identified easily. The profile extract section 
222 extracts the profile configuration of the part of the hand the body which exists in the 
image of the result calculated by the image difference operation part 221, i.e., before moving, 
and after moving. As a concrete example of an approach, a profile configuration can be easily 
extracted by extracting the edge of an image. 

[0055] It judges whether the form status change-ized discernment section 225 is the 
configuration of a fist where it is shown in drawing 10 (b) again whether it is the 
configuration of 1 finger as carries out detail discernment of the profile configuration of the 
part of the hand after moving, for example, the configuration shows to drawing 10 (a) 
extracted by the profile extract section 222. The location detecting element 224 calculates the 
barycentric coordinates of the profile configuration of the part of the hand of the user after 
[ said ] moving to coincidence. 

[0056] The icon generation section 24 generates the icon image which should be displayed on 
a display based on the discernment result of the configuration of the hand by the form status 
change-ized discernment section 225. The icon of an arrow head if the discernment result of 
the configuration of a hand is the configuration of 1 finger as an example of an icon image, as 
shown, for example in drawing 13 (a) is generated, and if it is the configuration of a fist, the 
icon of X mark as shown, for example in drawing 13 (b) will be generated. Moreover, it is good 
also as a configuration which generates the icon of the configuration which imitated the 
configuration of 2 fingers as shown, for example in drawing 13 (c) when the discernment 
result of the configuration of a hand was the configuration of 2 fingers, and generates the icon 
of the configuration which imitated the configuration of a fist as shown, for example in 
drawing 13 (d) when it was the configuration of a fist. 

[0057] A display and control section 23 controls the display position on the display 2 of the 
icon generated by the icon generation section 24, and consists of the coordinate 
transformation section 231 and the coordinate pars inflexa 232. The coordinate 
transformation section 231 performs conversion on the display coordinate of a display 2 of the 
picturized image from a coordinate, and the coordinate pars inflexa 232 reverses the 
right-and-left location of the changed display coordinate. 

[0058] Namely, location detecting element 224 Conversion on the display coordinate of a 
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display 2 from the barycentric coordinates in the image of the part equivalent to the hand of 
the user who detected is performed, a coordinate on either side is reversed, and an icon is 
displayed on a display 2. By this actuation, if a user moves a hand to the right, the same with 
having copied this actuation in the mirror, an icon will go on the display screen and it will 
move to the right. 

[0059] The example of actuation by this example constituted as mentioned above is shown 
below. As shown in drawing 8 , a user faces to the device equipped with the interface device by 
this example, and if it moves by making a hand into the configuration of 1 finger, the 
arrow-head cursor displayed on the display will move to the location of the arbitration 
corresponding to a motion of a hand. Next, when arrowhead cursor is moved, a hand is 
grasped and it is made a fist configuration by moving a hand after the virtual switch 
204,205,206 of the arbitration displayed on the display 2, it is the virtual switch 204,205,206. 
It is chosen and a command can be given to a host computer 1. 

[0060] In addition, although considered as the configuration which recognizes the 
configuration of the body in the picturized image, and a motion in this example, it cannot be 
overemphasized that it is good also as a configuration which recognizes the configuration of 
the body in the picturized image or either of the motions. 

[0061] the configuration of the body in the image which was picturized as mentioned above 
according to this example and or with the recognition section of operation which 
recognizes a motion the configuration of the body recognized by the recognition section of 
operation - and - or with the display which displays a motion The criteria image memory 
which accumulates the image picturized to the time amount before the image saved in the 
frame memory which saves the image picturized in the image pick-up section, and the frame 
memory as a criteria image is prepared. In the recognition section of operation The image in a 
frame memory, By extracting a difference with the criteria image accumulated into criteria 
image memory If a user goes to the image pick-up section, for example, directions are given 
according to a gesture, the given gesture can choose the virtual switch which it was displayed 
on the display screen, for example, was displayed on the display screen according to a gesture, 
and will not need input units, such as a mouse, but actuation of a very simple device of it will 
be attained. 

[0062] (The 3rd example) The appearance of the 3rd example of the interface device by this 
invention is the same as that of drawing 8 shown in the 2nd example, and only a part which 
explains the same part as the 2nd example using drawing 8 and drawing 10 , in addition is 
different is shown after drawing 14 . 

[0063] Drawing 14 shows the detail block diagram of the 3rd example. The image inputted 
from CCD camera 3 is stored in a frame memory 31. It judges whether the recognition section 
32 of operation is the configuration of a fist where the part which is equivalent to a user's 
hand, for example is extracted out of the image stored into the frame memory 31, and the 
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configuration shows drawing 10 (b) whether it is the configuration of 1 finger as shown in 
drawing 10 (a). 

[0064] Drawing 15 shows the detail of the recognition section 32 of operation, and explains the 
detail actuation using drawing 16 - drawing 20 . First, the profile extract section 321 The 
profile configuration of the body which exists in an image is extracted. A profile configuration 
can be easily extracted by carrying out binarization of the image and extracting the edge as a 
concrete example of an approach. Drawing 17 (a) is an example of the extracted border line, 
and the part of a hand shows the configuration of 1 finger. 

[0065] Next, as a border line is followed in the direction of the arrow head of drawing 
(counterclockwise rotation) with the start point s as the starting point in drawing and the 
profile configuration of the body extracted by the profile extract section 321 as shown in 
drawing 17 (a) is shown in drawing 19 , the profile wave operation part 322 The include angle 
theta from the horizontal line of the border line in each point x on a border line is extracted as 
a function to the distance 1 from a start point s. The coordinate of each point on the border line 
corresponding to distance 1 is used as a table, and it saves at a translation table 324 at the 
same time it changes wholly the distance 1 as shown in drawing 17 (b) into a wave 
configuration the bottom as a time -axis. The configuration filter 1 and the configuration filter 
2 which are shown in 3231 and 3232 are a filter which passes the band corresponding to the 
irregularity of the part of the hand in the profile wave shown in drawing 17 (b), respectively, 
and the irregularity of the part of a finger. 

[0066] With the configuration filter 1, as shown in drawing 17 (c), drawing 17 (b) It is changed 
into the smooth configuration wave only reflecting the irregularity of the part of a hand. With 
moreover, the configuration filter 2 As shown in drawing 17 (d), it is changed into the 
configuration wave only reflecting the irregularity of a finger, and differentiates with 
differentiators 3251 and 3252, respectively and, finally the configuration differential wave 
shown in drawing 17 (e) and drawing 17 (f) is acquired. Judging whether the configuration 
judging section 3262 is the configuration of a fist where it is shown in drawing 10 (b) again 
whether the profile configuration of the part of a hand is a configuration of 2 fingers as shown 
in drawing 10 (a). the coordinate operation part 3261 calculates the barycentric coordinates of 
the profile configuration of the part said user's hand to coincidence. The coordinate operation 
part 3261 calculates the location where a big pulse shape in the configuration differential 
wave shown in drawing 17 (e) exists, and lcl and lc2, and is the coordinate transformation 
table 324. It changes into cl point shown in drawing 20 , and c2 point, and from the border 
line of the part of the hand from cl point to c2 point, the center of gravity of the part of a hand 
is calculated and it outputs as a hand coordinate. 

[0067] Moreover, the configuration judging section 3262 counts and outputs the number of the 
pulse shape with which the part of the finger in the configuration differential wave of drawing 
17 (£) corresponds. That is, since two big pulse shape equivalent to the part of a finger exists, 



JP09- 185456 
Translated by Computer 

18 

in the case of drawing 17 (f), it judges with it being the configuration of 2 fingers as shown in 
drawing 10 (a), and outputs to it. As shown in drawing 18 (a), in the case of the configuration 
which grasped the hand, there is almost no irregularity of the part of a finger. Moreover, the 
output of the configuration filter 2 As are shown in drawing 18 (c), and it becomes a 
configuration wave without irregularity, therefore the output of a differentiator 3262 is also 
shown in drawing 18 (d) It becomes the configuration differential wave which does not have 
pulse shape, and the count of pulse shape can be set to 0, and it judges with it being the 
configuration of a fist as shown in drawing 10 (b), and can output. A simple threshold 
approach, a neural network, etc. can be used as an example of a concrete configuration of this 
configuration judging section 3262. 

[0068] The icon generation section 34 of drawing 14 generates the icon image which should be 
displayed on a display based on the discernment result of the configuration of the hand by the 
configuration judging section 3262 in drawing 15 . For example, the icon of X mark if it is the 
configuration of a fist again, as shows the icon of an arrow head if the discernment result of 
the configuration of a hand is the configuration of 1 finger, as shown, for example in drawing 
16 (a) to drawing 16 (b), for example is generated. A display and control section 33 controls the 
display position on the display of the icon generated by the icon generation section 34, and is 
the coordinate transformation section 331. It consists of the coordinate pars inflexa 332. The 
coordinate transformation section 331 performs conversion on the display coordinate of a 
display of the picturized image from a coordinate, and is the coordinate pars inflexa 332. The 
right-and left location of the changed display coordinate is reversed. That is, conversion on 
the display coordinate of a display from the barycentric coordinates in the image of the part 
equivalent to a user's hand which the coordinate operation part 3261 in drawing 15 outputted 
is performed, a coordinate on either side is reversed, and an icon is displayed on a display. By 
this actuation, if a user moves a hand to the right, the same with having copied this actuation 
in the mirror, an icon will go by the shape of the display screen, and it will move to the right. 
[0069] The example of actuation of this example constituted as mentioned above is shown 
below. Icon 203 displayed on the display 2 when the user faced to the device equipped with the 
interface device by this example and it moved by making a hand into the configuration of 1 
finger, as shown in drawing 8 Arrow-head cursor moves to the location of the arbitration 
corresponding to a motion of a hand. Next, virtual switch 204,205,206 of the arbitration 
displayed on the display 2 If arrowhead cursor is moved, a hand is grasped and it is made a 
fist configuration by moving a hand upwards, the virtual switch is chosen and a command can 
be given to a host computer 1. 

[0070] Moreover, if the configuration of a hand itself is iconified as an example of the icon to 
display as shown in drawing 16 (c) and (d), a motion and correspondence of an actual hand 
can be taken and it is intelligible. The profile configuration data of the hand which could 
register beforehand the image as shown in drawing 16 (c) and (d), and was specifically 
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extracted by the profile extract section can be reduced or expanded to the magnitude of 
arbitration, and can also be used as an icon image. 

[0071] Thus, as the include angle of the border line to a horizontal line, i.e., a profile wave, if a 
user goes to the image pick-up section of an interface device, for example, this example gives 
directions according to a gesture, the image pick up section will picturize a user's image, will 
extract the profile of a user's image, and will be changed by setting an axis of abscissa as the 
die length of the border line which made the reference point on a profile the start point. This 
profile wave is changed into the configuration wave which expresses the shape of toothing of a 
finger with the configuration filter constituted with the band pass filter of a predetermined 
band, for example, the band pass filter equivalent to the irregularity of a finger, while it 
calculates a location with a hand, only counts the pulse number which exists in this 
configuration wave, and can make the number of the finger currently sent, i.e., the 
configuration of a hand, judge correctly. The virtual switch which the given gesture was 
displayed on the display screen based on a location and a configuration of this sort, for 
example, was displayed on the display screen cannot be chosen according to a gesture, and 
input units, such as a mouse, are not needed, but actuation of a very simple device is attained. 
[0072] (The 4th example) The 4th example of this invention is explained with reference to a 
drawing. Drawing 26 shows the block diagram of the 4th example of an interface device. 
[0073] drawing 26 ■■ setting - 41 - an input means and 42 - a cursor storage means and 43 - 
a body storage means and 44 - a display means and 45 - an interaction judging means and 
45a - a distance count means and 45b - for a movement vector calculation means and 45e, a 
configuration judging means and 45f of a study means and 45g are [ a recognition means of 
operation and 45c / a comprehensive judgment means and 45d / a coordinate transformation 
means and 46 ] look input means. 

[0074] In drawing 26 , an operator operates it to the input means 41, and it is based on said 
actuation result. The representation point coordinate and the configuration where the cursor 
storage means 42 represents the location in the virtual space of cursor are changed and 
memorized. The display means 44 is based on the representation point coordinate and the 
configuration of representing the location in the virtual space of the virtual body which the 
representation point coordinate and the configuration of representing the location in the 
virtual space of the cursor which the cursor storage means 42 memorizes, and the body 
storage means 43 memorize. Said cursor and said virtual body It two-dimensional-displays or 
displays in three dimensions. 

[0075] The look input means 46 detects the location of the look of the operator on said display. 
Distance count means 45a calculates the distance in the virtual space of cursor and a virtual 
body by being based on the coordinate of the representation point memorized for the cursor 
storage means 42 and the body storage means 43. Of operation recognition means 45b 
recognizes actuation actuation based on the contents memorized for the cursor storage means 
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42 and the body storage means 43. 45d of movement vector calculation means computes the 
migration direction and migration length of cursor in a virtual space. It judges whether 
configuration judging means 45e is suitable for the configuration of cursor and the 
configuration of a virtual body to cause an interaction. 45f of study means memorizes the 
location of a said cursor and said virtual body when comprehensive judgment means 45c 
causes the interaction between cursor and a virtual body, and the relation of a configuration, 
and they output whether the current condition is similar to the condition of having caused the 
interaction in the past. 

[0076] The distance of a between [ the cursor to which distance count means 45a outputs 
comprehensive judgment means 45c, and a virtual body ], The recognition result which of 
operation recognition means 45b has recognized, and the migration direction and migration 
length of cursor which 45d of movement vector calculation means computes, The location of 
the look which the look input means 46 detects, and the judgment result of configuration 
judging means 45e, It judges whether said cursor and said virtual body interact based on a 
similar degree with the interaction of the past which 45f of study means outputs, and said 
representation point coordinate and configuration of cursor or said virtual body are changed 
according to the result of an interaction. When the interaction judging means 45 causes an 
interaction, 45g of coordinate transformation means performs coordinate transformation so 
that both location may approach the coordinate in the virtual space of the cursor and the 
target virtual body which distance count means 45a uses for distance count. 
[0077] Drawing 22 (a) and (b) are the things of the manipulator configuration of two fingers 
with the gestalt of operation of the first of the cursor used for the interface device of this 
invention. In drawing 22 , the condition to which (a) opened the finger, and (b) are in the 
condition which closed the finger. Drawing 22 (c) and (d) are the things of the manipulator 
configuration of 2 finger 2 joint with the gestalt of operation of the second of the cursor used 
for the interface device of this invention. In drawing, the condition to which (c) opened the 
finger, and (d) are in the condition which closed the finger. Drawing 22 (e) and CO are the 
gestalten of operation of the third of the cursor used for the interface device of this invention, 
and are the thing of the shape of a note of five fingers. In drawing 22 , the condition to which 
(e) opened the hand, and (f) are in the condition which closed the hand. Drawing 23 (a) and (b) 
are the examples of the body in the virtual space used for the interface device of this invention 
((a) is a cube and (b) is a flat- surface object). 

[0078] Actuation of the interface device constituted as mentioned above is explained. This 
example assumes the actuation which moves cursor like drawing 22 into the virtual space of a 
three dimension, and an operator holds a virtual body like drawing 23 which exists in said 
virtual space, and moves. Actuation of an operator is performed to the input means 41. 
[0079] Here, the command input by a means or a camera as shown in drawing 27 (a) * (c), a. 
keyboard, speech recognition, etc. can be used for an input as an input device which inputs 
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the information for changing the location or configuration of cursor. Drawing 27 (a) is a mouse 
and operates cursor by migration of a mouse body, the click of a carbon button, etc. Drawing 
27 (b) is a data glove, equip people's hand with it, makes the location in the real space of the 
joint angle of a finger, or a data glove reflect in the location and configuration of cursor, and 
operates cursor. Drawing 27 (c) is a joy stick and operates cursor according to actuation of a 
lever, and concomitant use with a manual operation button. In using a camera, a part of body 
or body (hand etc.) is picturized with a camera, and it reads the configuration and location of a 
hand. 

[0080] Drawing 28 shows the gestalt of 1 implementation of the configuration extract at the 
time of picturizing only a hand with a camera. Drawing 28 (a) is the example which picturized 
the hand with the camera. It is drawing 28 (b) which carried out binarization of the 
brightness of each pixel of the image of drawing 28 (a). In drawing 28 (b), the judgment of the 
degree of closing motion of a hand is attained with the ratio of the die length of the side of 
rectangular length and the die length of the horizontal side which are circumscribed to a **** 
field etc., and the input of a location and distance is attained from the barycentric coordinates 
and area of the whole black pixel. The input means 41 outputs said contents of actuation (the 
movement magnitude of cursor, the amount of modification of a cursor configuration, etc.) to 
the cursor storage means 42. 

[0081] The cursor storage means 42 memorizes the coordinate and configuration in the virtual 
space of the representation point of the cursor memorized with a cursor storage means based 
on the contents of actuation which the input means 41 outputted. As a representation point, 
the barycentric coordinates (X0, Y0, Z0) of cursor are used as a representation point. 
[0082] In addition, as a representation point, a main coordinate of each side, a coordinate of 
top-most vertices, etc. which constitute cursor may be used. In the case of drawing 22 (a), in 
the spacing d of two fingers, drawing 22 (b), and (c), let as a configuration interior angle 
thetan (the number whose n is a joint: when thetan becomes small, the condition that a joint 
bends is shown) of the joint of each finger be storage information. In addition, as a 
configuration, the fingertip of each finger, the coordinate in the virtual space of each joint, etc. 
may be used. 

[0083] The body storage means 43 memorizes the coordinate and configuration of a 
representation point of a virtual body in the virtual space shown in drawing 23 set as the 
object of actuation. As a representation point, they are the barycentric coordinates (cube: (Xl, 
Yl, Zl), fiat-surface object : (X2, Y2, Z2)) of a virtual body. 

It uses as a representation point. In addition, as a representation point, a main coordinate of 
each side, a coordinate of top most vertices, etc. which constitute a virtual body may be used. 
The parameter alpha showing the configuration beforehand defined as a configuration is 
memorized as a configuration (a cube is defined as alpha= 1 and a flat* surface object is 
defined as alpha= 2 here). In addition, a top -most* vertices coordinate etc. may be used as a 
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configuration. 

[0084] The display means 44 indicates the image at which it looked from the view which 
specified the virtual space beforehand based on the cursor which the cursor storage means 42 
and the body storage means 43 memorize, the location of a virtual body, and the information 
on a configuration by two-dimensional. The example of a display of a display means is shown 
in drawing 29 (a). If an operator operates it, the display position or configuration of cursor will 
change and an operator will continue actuation based on the display. 

[0085] The interaction judging means 45 judges whether cursor has held the body, whenever a 
cursor location changes (existence of an interaction), and when cursor judges that the body 
has been held, it also moves the coordinate of a virtual body in accordance with migration of 
cursor. Distance count means 45a calculates the barycentric coordinates (XI, Yl, Zl) of the 
virtual body memorized for the barycentric coordinates (X0, Y0, Z0) and the body storage 
means 43 of the cursor memorized for the cursor storage means 42, and the distance of (X2, 
Y2, Z2). 

[0086] An of operation recognition means 45b means recognizes actuation of "holding" as 
actuation registered beforehand, using change of the configuration of cursor. It is recognized 
as actuation and recognized as the actuation "holds" the condition the condition that the 
spacing d of two fingers continues decreasing "is held" in the case of the cursor of drawing 22 
(a) that include -angle thetan of all fingers continues decreasing in the case of the cursor of 
drawing 22 (b) and (c). In addition, a serial change of the parameters (said d, thetan, etc.) with 
which a configuration is expressed as the recognition technique of operation may be used as 
the recognition technique, after making specific actuation learn beforehand using the time 
series pattern recognition technique (table matching, DP matching, a HIDOUN Markov 
model (HMM), RIKARENTO neural network, etc.). 

[0087] 45d of movement vector calculation means computes the migration direction and 
movement magnitude of cursor in a virtual space by using change of the barycentric 
coordinates (X0, Y0, Z0) of cursor, for example, barycentric coordinates (X0, Y0, Z0) t of the 
present time of day t Barycentric coordinates t-1 in front of 1 time of day (X0, Y0, Z0) 
difference - let the direction and magnitude of a vector be the migration direction and 
movement magnitude of cursor. 

[0088] Configuration judging means 45e judges (whether the configuration of cursor is 
suitable to cause a virtual body and an interaction) for whether it is a configuration with the 
suitable configuration of the cursor memorized for a cursor storage means to hold the virtual 
body of the configuration memorized for a body storage means. Here, when the value of the 
parameter alpha showing an objective configuration is 1, the condition that the finger of 
cursor is open is made into a suitable condition. In the case of the cursor of drawing 22 (a), as 
decision of the condition that the finger of cursor is open, the value of d is the maximum dmax 
of d, for example. It considers as the case of being larger than the mean value of 0, and, in 
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drawing 22 (b) and (c), each joint angle thetan considers all [ mean value / each / of maximum 
the tan max and 0 ] as the case of being large. 

[0089] When the value of the parameter alpha showing an objective configuration is 0, the 
condition that spacing of the fingertip of cursor is narrow is made into a suitable condition. In 
the case of the cursor of drawing 22 (a), as decision of the condition that the fingertip of cursor 
is narrow, the value of d is the maximum dmax of d, for example. It considers as the case of 
being smaller than the mean value of 0, and, in drawing 22 (b) and (c), each joint angle thetan 
is each maximum thetan max. All [ mean value / of 0 ] are considered as the case of being 
small, in addition - if the parameter (d or thetan) showing the cursor configuration when 
having held in the condition that cursor contacts a virtual body in a virtual space as a 
diagnosis of a configuration is memorized beforehand and it is in agreement in the range 
whose value of each parameter is **30% It holds. It can perform judging that it is suitable for 
actuation etc. 

[0090] The look input means 46 detects an operator's look, and computes the coordinate 
(coordinate of a fixation point) at which an operator gazes on the screen which the display 
t means 44 displays. The fixation point on said screen is computed by measuring the location of 

an operator's head using a camera etc. using the approach of detecting the sense of an 
operator's pupil, using a CCD camera, a photosensor, etc. as a detection means of a look etc. 
[0091] When, as for 45f of study means, cursor judges [ comprehensive judgment means 45c ] 
that the virtual body has held The relative physical relationship (vector which connects the 
center of gravity of cursor and the center of gravity of a virtual body) of the location of the 
parameter (d or thetan) showing the configuration of ** cursor, the parameter alpha showing 
the configuration of the held virtual body, and said cursor and the location of said virtual body 
is memorized. The physical relationship of the center of gravity of the parameter showing the 
configuration of a current virtual body, the parameter showing the configuration of a 
surrounding virtual body, and current cursor and the center of gravity of a surrounding 
virtual body When the body has been held in the past, in being near, it judges that it is similar 
with the past situation, and outputs 1 (for example, when the value of the element showing 
each parameter and physical relationship of each dimension of a vector is in agreement in the 
past value and **30% of range etc.), and in the case of others, 0 is outputted. In addition, the 
relative physical relationship of the location of the parameter showing the configuration of the 
cursor in the case of having held the body, the parameter alpha showing the configuration of 
the held virtual body, and said cursor and the location of said virtual body may be learned 
using a neural network etc. in the past as other study means. Moreover, it can also learn 
including the physical relationship of the fixation point coordinate on the screen detected with 
the look detection means 46 as an item to learn, and the coordinate on the display screen of 
cursor etc. 

[0092] When the body has been held, coordinate transformation of the 45g of the coordinate 
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transformation means is carried out so that the distance of the cursor in a virtual space and 
the target virtual body may approach the coordinate which a distance count means uses for 
distance count (when an interaction is caused). For example, when each coordinate when 
cursor has held the virtual body is (100, 150, 20), and (105, 145, 50), a coordinate 
transformation means changes like [ Z coordinate / of a difference / largest ] (l) type among 
each axis of coordinates. 

[0093] Z'=0.8xz [ ] (l) - here, the Z coordinate of the center of gravity of the cursor which, 

as for z, a coordinate transformation means considers as an input, and a virtual body, and Z' 
show the Z coordinate which a coordinate transformation means outputs. 
[0094] In this case, the value of X coordinate and the value of Y coordinate are not changed. 
Moreover, since the value memorized for a cursor storage means and a body storage means 
does not change, the screen which a display means displays does not change. If actuation 
which an operator holds is performed by performing the above conversion even when the 
distance in a virtual space is separated, the cursor in the case of performing distance count 
and the distance between virtual bodies will become small after it, and a distance count 
^ means will become possible [ calculating the distance near the depth perception which an 

operator perceives ]. 

[0095] When comprehensive judgment means 45c is below the criteria that the distance of a 
between [ the cursor which distance count means 45a outputs, and a virtual body ] defined 
beforehand, When of operation recognition means 45b has recognized the actuation registered 
beforehand of "holding" It judges with the interaction of "holding" having occurred and the 
value of the barycentric coordinates of the virtual body which is memorized for the body 
storage means 43 until an interaction is completed and which is held "is held" henceforth is 
made in agreement with the barycentric coordinates of cursor. Here, a value with actually 
larger cursor and a body than the distance which can contact in a virtual space is sufficient as 
said reference value defined beforehand. For example, the actuation which an operator will 
hold if the distance of a virtual body and cursor is below the reference value of said distance in 
the case of drawing 25 (arrangement of drawing 24 ) is directed for the input means 1, and if 
the actuation which of operation recognition means 45b holds is recognized, it will become 
possible to hold a virtual body and to move. 

[0096] Moreover, when, as for comprehensive judgment means 45c, two or more virtual bodies 
exist below in the criteria of said distance, It is aimed only at the body below criteria (for 
example, 90 degrees) with the include angle of the segment (wavy line) which ties cursor and 
a virtual body as shown in drawing 29 (b), and the migration direction (arrow head) of the 
cursor which 45d of movement vector calculation means computes to make. The judgment of 
an interaction which took the migration direction of cursor into consideration in actuation of 
an operator is attained (what is most located in the upper part among 3 bodies by a diagram is 
chosen). 
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[0097] Moreover, about the migration length of cursor, when migration length is larger than 
the migration length criteria defined beforehand, an interaction is not caused. Thereby, when 
cursor is only being moved, it becomes possible not to cause the interaction which an operator 
does not mean. 

[0098] Moreover, as shown in drawing 29 (c), when two or more virtual bodies are meeting 
said criteria, comprehensive judgment means 45c sets the virtual body near the location of the 
fixation point which the look input means 46 detects as the object "to hold" (the body of the 
left-hand side near "+" mark which means a fixation point by a diagram is chosen). This 
becomes possible to choose an object easily using an operator's look. 

[0099] Moreover, as shown in drawing 29 (d), when the body which approached on the screen 
exists, comprehensive judgment means 45 c sets the virtual body which is in agreement with 
the configuration of the cursor which configuration judging means 45e judged as the object "to 
hold" (since spacing of the finger of cursor is narrow by a diagram, it judges that a flat-surface 
object is suitable as an object of the actuation "to hold", and a flat- surface object chooses). This 
is enabled to choose the virtual body which an operator means with the configuration of cursor, 
and actuation becomes easy by matching the cursor configuration of which it is easy to be 
reminded when an operator holds a virtual body. 

[0100] Moreover, comprehensive judgment means 45c chooses preferentially the virtual body 
it was judged that was similar with the case where 45f of study means has held the body in 
the past. Thereby, an operator reproduces decision near the actuation performed in the past, 
and becomes possible [ improving operability ]. 

[0101] Thus, this example can raise operability in the interface which performs an interaction 
with the virtual body using the cursor in a virtual space by determining it based on the 
example of the actuation in actuation of an operator, a look, or the past rather than 
determining the existence of generating of the interaction of the cursor which an operator 
operates in a virtual space, and the virtual body in a virtual space only in the distance of 
cursor and a virtual body. 

[0102] In addition, although this example explained using the actuation which holds a virtual 
body, using cursor as an interaction, the treatment same about actuation of the directions 
(pointing) to a virtual body, a collision and friction, a blow, remote operation, etc. otherwise is 
possible. Moreover, the same effectiveness is acquired, when a virtual space is 
two-dimensional space, or also when a display means uses a three-dimension three 
dimentional display display etc. Moreover, you may realize using the software on a computer, 
using hardware as an implementation means. 

[0103] This example the existence of generating of the interaction of the cursor which an 
operator operates in a virtual space, and the virtual body in a virtual space as mentioned 
above Only the distance between components does not determine said cursor and said virtual 
body in a virtual space. When a comprehensive judgment means judges the existence of 



JP09- 185456 
Translated by Computer 

26 

generating of an interaction by actuation of the cursor which the distance and the recognition 
means of operation between the representation points which a distance count means 
computes recognize It becomes possible to cause an interaction also to the object with which 
the distance in a virtual space is not necessarily close, and it becomes possible to offer the 
good input/output interface of operability. Moreover, since it is not necessary to calculate all 
the distance between components for said cursor and said virtual body in a virtual space like 
the conventional contact diagnosis, computational complexity is mitigated, and it becomes 
possible to attain improvement in the speed of processing. 
[0104] 

[Effect of the Invention] This invention can control easily the information displayed on the 
screen by the configuration of a hand, and the motion in the information displayed in the 
screen with sufficient operability as mentioned above by indicating by cursor on a screen by 
making into a special configuration the description of the configuration of the configuration of 
an operator's hand, or the hand which has recognized the motion again and has been 
recognized. 

[0105] Furthermore, while displaying on a screen and indicating by cursor by making the 
description of the configuration of a hand into a special configuration, the interface whose 
operability of actuation of directing or holding a display object improved further is realizable 
by making serially automatically the judgment of an interaction which met the intention of an 
operator in relation with display objects other than the cursor display. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing ll The external view of the interface device by the 1st example of this invention 
[Drawing 2] The detail block diagram of the interface device by the 1st example of this 
invention 

[Drawing 31 (a) - (c) Drawing showing the example of the configuration of the hand judged 
with this interface device 

[Drawing 41 Drawing showing the example of a detail of the configuration discernment means 
of the interface device by this example 

[Drawing 5] Drawing showing an example of the operation by the image difference operation 
part of this example 

[Drawing 61 (a) - (d) Drawing showing the example of the icon which the icon generation 
section of this example generates 

[Drawing 7l (a) - (c) External view showing the example of the interface device by this 
example of operation 

[Drawing 81 The external view of the interface device by the 2nd example of this invention 
[Drawing 91 The detail block diagram of the interface device by the 2nd example of this 
invention 

[Drawing 101 (a) and (b) Drawing showing the example of the configuration of the hand 
judged with this interface device 

[Drawing 111 Drawing showing the detail example of the recognition section of the interface 
device of this example of operation 

[Drawing 121 Drawing showing an example of the operation by the image difference operation 
part of this example 

[Drawing 131 (a) - (d) Drawing showing the example of the icon which the icon generation 
section of this example generates 

[Drawing 141 The detail block diagram of the interface device by the 3rd example of this 
invention 

[Drawing 15] Drawing showing the recognition section of the interface device of the 3rd 
example by this invention of operation 

[Drawing 161 (a) - (d) Drawing showing the example of the icon displayed on the display 
screen with the interface device by this example 

[Drawing 17] (a) - (0 Drawing showing actuation of the recognition section of the interface 
device by this example of operation 

[Drawing 18] (a) - (£) Drawing showing actuation of the recognition section of the interface 
device by this example of operation 

[Drawing 19] Drawing showing actuation of the recognition section of the interface device by 
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this example of operation 

[Drawing 201 Drawing showing actuation of the recognition section of the interface device by 
this example of operation 

rPrawing 21.1 The block diagram showing the interface device in the 4th example of this 
invention 

[Drawing 221 (a) Drawing showing the condition that cursor opened in an example of the 
cursor used for the interface device of this example 

(b) Drawing showing this closing ****** 

(c) Drawing showing the condition that cursor opened in other examples of the cursor used for 
the interface device of this example 

(d) Drawing showing this closing ****** 

(e) Drawing showing the condition that cursor opened in the example of further others of the 
cursor used for the interface device of this example 

(£) Drawing showing this closing ****** 

[Drawing 231 (a) Drawing showing the form of an example of the virtual body used for the 
interface device of this example 

(b) Drawing showing the form of other examples of this virtual body 

[Drawing 241 (a) The front view showing the cursor of a virtual space, and arrangement of a 
virtual body 

(b) The side elevation showing the cursor of a virtual space, and arrangement of a virtual body 
[Drawing 25] Drawing showing the example of a display of the virtual space for explaining 
this example 

[Drawing 26] The block diagram showing an example of the interface device of this example 
[Drawing 27] (a) Drawing showing an example of the input unit in the input means used for 
the interface device of this example 

(b) Drawing showing other examples of the input unit in the input means used for the 
interface device of this example 

(c) Drawing showing the example of further others of the input unit in the input means used 
for the interface device of this example 

[Drawing 28] (a) Drawing showing an example of the image which picturized the hand using 
the camera of this example 

(b) Drawing showing an example which carried out binarization of the image which picturized 
the hand using the camera of this example 

[Drawing 29] (a) Drawing showing an example of the screen which the display means used for 

the interface device of this example displays 

(b) Drawing showing the second example of this display screen 

Cc) Drawing showing the third example of this display screen 

(d) Drawing showing the fourth example of this display screen 
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[Drawing 30l Drawing showing the conventional interface device 
[Description of Notations] 

1 Host Computer 

2 Display 

3 CCD Camera 

21 Frame Memory 

22 Configuration Discernment Means 

23 Display and Control Section 

24 Icon Generation Section 

25 Criteria Image Memory 

26 Renewal Section of Criteria Image 

31 Frame Memory 

32 Recognition Section of Operation 

33 Display and Control Section 

34 Icon Generation Section 

41 Input Means 

42 Cursor Storage Means 

43 Body Storage Means 

44 Display Means 

45 Interaction Judging Means 
45a Distance count means 

45b Recognition means of operation 
45c Comprehensive judgment means 
45d Movement vector calculation means 
45e Configuration judging means 
45f Study means 

45g Coordinate transformation means 
200 Icon 
201,202 Menu 
203 Icon 

204,205,206 Virtual switch 

221 Image Difference Operation Part 

222 Profile Extract Section 

223 Configuration Discernment Section 

224 Location Detecting Element 

225 Form Status Change-ized Discernment Section 

231 Coordinate Transformation Section 

232 Coordinate Pars Inflexa 



261 Timer 

262 Renewal Section of Image 

321 Profile Extract Section 

322 Profile Wave Operation Part 
324 Coordinate Table 

331 Coordinate Transformation Section 

332 Coordinate Pars Inflexa 

3231 Configuration Filter 1 

3232 Configuration Filter 2 

3251 Differentiator 

3252 Differentiator 

3261 Coordinate Operation Part 

3262 Configuration Judging Section 
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